Skip to content

Wrap the live ICudaEngine after compile instead of serialize/deserialize. - #4698

Open
micwill755 wants to merge 1 commit into
network-constructfrom
convert-finalize
Open

micwill755 wants to merge 1 commit into
network-constructfrom
convert-finalize

Conversation

@micwill755

Copy link
Copy Markdown
Collaborator

Summary

  • After build_engine_with_config, in-process convert_module wraps the live ICudaEngine instead of serialize() + C++ deserializeCudaEngine.
  • interpret_module_to_result(..., serialize_engine=False) keeps cuda_engine on SerializedInterpreterResult. Cache insert, convert_exported_program_to_serialized_trt_engine, and get_extra_state / save still serialize from the live engine.
  • Cross-compile-for-Windows still serializes at interpret time. In-process compile uses Python TRTEngine.execute() until save/reload (C++ execute path needs bytes).
  • Klein NVFP4 cold compile: ICudaEngine.serialize 1.15s → 0, setup_engine 2.40s → 0.001s, compile wall 36.95s → 33.17s. First image ~1.12s and steady ~0.43s unchanged.

PR 10 of the compile-startup stack; based on #4693.

Type of change

  • Performance (non-breaking for in-process compile; save/export still produce serialized engines)

Test plan

  • tests/py/dynamo/runtime/test_live_engine_wrap.py (compile tiny Add: serialized_engine is None, live wrap, cosine match, get_extra_state serializes)
  • Klein NVFP4 cold compile (torchtrt-213rc10): serialize 1.15s and setup_engine 2.40s gone; first image and steady unchanged
  • Existing dynamo runtime tests (CI)

Checklist

  • Code follows project style (pre-commit: isort, ruff, black, mypy)
  • Self-review
  • Comment on why the skip is valid
  • Docs update (not needed)
  • New tests
  • Labels

Stack created with GitHub Stacks CLI • Give Feedback 💬

@meta-cla meta-cla Bot added the cla signed label Sep 9, 2026
@micwill755
micwill755 added this pull request to stack #4533 September 9, 2026 08:10
@github-actions github-actions Bot added component: tests Issues re: Tests component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: api [Python] Issues re: Python API component: runtime component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Sep 9, 2026
@github-actions
github-actions Bot requested a review from cehongwang September 9, 2026 08:10
…ize.

In-process convert_module keeps the builder engine and skips the bytes roundtrip. Cache, export, and save still serialize.
@cehongwang

Copy link
Copy Markdown
Collaborator

This PR needs some work. I don't think it can be done before the branch cut. Can you do a benchmark for me first?

Choose a few models and record:

  1. compilation time
  2. peak host memory usage (offload cpu =True)
  3. peak CUDA memory usage (offload cpu =True)

I want to know the impact before moving forward with this PR. We meticulously designed the format of interpret_result and lowered the memory usage as much as possible, but this PR seems to mix it up. Also it mixes up the usage of Python Engine and C++ runtime engine as well.

@cehongwang cehongwang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As described above

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: runtime component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants